<html>
<head><meta charset="utf-8"><title>Live unit testing · t-compiler/rust-analyzer · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/index.html">t-compiler/rust-analyzer</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html">Live unit testing</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="217883005"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217883005" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eggyal <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217883005">(Nov 25 2020 at 13:21)</a>:</h4>
<p>Not sure whether this is the right place to ask—if not please do point me elsewhere!  Just wondering whether there are any efforts underway (or if not, whether it is on any roadmap) to implement something akin to "live unit testing" for Rust (where the IDE continuously updates visual indicators of unit test results AS ONE CODES: see for example <a href="https://wallabyjs.com/">https://wallabyjs.com/</a> and <a href="https://docs.microsoft.com/en-us/visualstudio/test/live-unit-testing">https://docs.microsoft.com/en-us/visualstudio/test/live-unit-testing</a> )?  Compilation time obviously a hinderance but perhaps moving toward a realistic possibility with <a href="https://github.com/bjorn3/rustc_codegen_cranelift/issues/1087">https://github.com/bjorn3/rustc_codegen_cranelift/issues/1087</a> ... cc <span class="user-mention" data-user-id="133247">@bjorn3</span></p>



<a name="217883766"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217883766" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217883766">(Nov 25 2020 at 13:28)</a>:</h4>
<p>Seems like this would require running the compiler on not-yet-saved files, aka VFS support</p>



<a name="217883904"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217883904" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jeremy Kolb <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217883904">(Nov 25 2020 at 13:29)</a>:</h4>
<p><a href="https://github.com/rust-analyzer/rust-analyzer/issues/3601">https://github.com/rust-analyzer/rust-analyzer/issues/3601</a></p>



<a name="217883988"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217883988" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217883988">(Nov 25 2020 at 13:30)</a>:</h4>
<p>and that, yeah :)</p>



<a name="217884064"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217884064" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eggyal <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217884064">(Nov 25 2020 at 13:31)</a>:</h4>
<p><span class="user-mention" data-user-id="211727">@Jonas Schievink</span>  Perhaps.  Or else mirroring the source files in an on-disk cache to which modifications (not yet saved to the working tree) are stored and compiling that (I believe this is how Wallaby does it, via IDE plugins).</p>



<a name="217884110"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217884110" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217884110">(Nov 25 2020 at 13:31)</a>:</h4>
<p>yeah, but that's pretty error-prone</p>



<a name="217884267"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217884267" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217884267">(Nov 25 2020 at 13:33)</a>:</h4>
<p>This is a nice idea! When using nightly rustc, VFS support is easy to implement. All you have to do is implement <a href="https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/source_map/trait.FileLoader.html"><code>FileLoader</code></a> and pass it to <code>rustc_driver</code> in a custom driver. This is pretty much what rls does as far as I know. As for the referenced cg_clif issue, I made some progress on lazy compilation. The same Cranelift changes necessary for this would enable hot code swapping from the Cranelift side. I don't know how easy it will be to abuse incremental compilation for this purpose though, but it is certainly high on my todo list.</p>



<a name="217884410"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217884410" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Diebold <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217884410">(Nov 25 2020 at 13:34)</a>:</h4>
<p>we don't call rustc in-process, so it's not so easy <span aria-label="grimacing" class="emoji emoji-1f62c" role="img" title="grimacing">:grimacing:</span></p>



<a name="217884471"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217884471" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217884471">(Nov 25 2020 at 13:34)</a>:</h4>
<p>If only rust-analyzer were to require nightly...</p>



<a name="217884624"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217884624" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jeremy Kolb <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217884624">(Nov 25 2020 at 13:36)</a>:</h4>
<p>Maybe someone could take <a href="https://github.com/rust-analyzer/rust-analyzer/pull/5765">https://github.com/rust-analyzer/rust-analyzer/pull/5765</a> and push it over the edge</p>



<a name="217884856"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217884856" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217884856">(Nov 25 2020 at 13:38)</a>:</h4>
<p>I don't see pinning r-a to nightly Rust versions as too big of a problem, as long as we only update the version when really necessary.</p>



<a name="217884893"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217884893" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217884893">(Nov 25 2020 at 13:39)</a>:</h4>
<p>(I feel like we update dependencies way too frequently, my target dir is over 30 GB already)</p>



<a name="217884977"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217884977" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eggyal <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217884977">(Nov 25 2020 at 13:40)</a>:</h4>
<p><span class="user-mention" data-user-id="203366">@Jeremy Kolb</span> the test explorers are great but do they update live?  I think this requires instrumentation in order to determine which tests have been affected by any given modifications, or else one will need to rerun the entire suite after every edit. Perhaps <span class="user-mention" data-user-id="296355">@Rich Kadel</span>'s work on <a href="https://github.com/rust-lang/rust/issues/79121">https://github.com/rust-lang/rust/issues/79121</a> could help here</p>



<a name="217885099"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217885099" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217885099">(Nov 25 2020 at 13:40)</a>:</h4>
<p>That would require pretty far-reaching code analysis</p>



<a name="217885224"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217885224" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eggyal <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217885224">(Nov 25 2020 at 13:42)</a>:</h4>
<p><span class="user-mention" data-user-id="211727">@Jonas Schievink</span>  Or just run tests (with instrumentation) and record what code regions each has exercised/covered</p>



<a name="217885381"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217885381" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jeremy Kolb <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217885381">(Nov 25 2020 at 13:43)</a>:</h4>
<p>I'm not sure but my guess is that it's probably not live.</p>



<a name="217885548"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217885548" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Diebold <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217885548">(Nov 25 2020 at 13:44)</a>:</h4>
<p>I'd say the pieces for this are coming together, but we're probably still a few years away from being able to do this</p>



<a name="217885842"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217885842" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eggyal <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217885842">(Nov 25 2020 at 13:47)</a>:</h4>
<p>Sounds like a github issue sketching out the design and dependencies could be helpful?</p>



<a name="217886001"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217886001" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217886001">(Nov 25 2020 at 13:48)</a>:</h4>
<p>heh, I suggested something like this a while back: <a href="#narrow/stream/242791-t-infra/topic/Speculative.20CI.3F/near/200533333">https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/Speculative.20CI.3F/near/200533333</a></p>



<a name="217886271"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217886271" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217886271">(Nov 25 2020 at 13:50)</a>:</h4>
<p>ugh, the official discord hid the #infra channel for some reason so you can't look at it anymore</p>



<a name="217886294"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217886294" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217886294">(Nov 25 2020 at 13:50)</a>:</h4>
<p>I had found a project doing this in python</p>



<a name="217886374"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217886374" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Diebold <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217886374">(Nov 25 2020 at 13:51)</a>:</h4>
<p><span class="user-mention" data-user-id="232545">@Joshua Nelson</span> I think in the context of CI this doesn't really help. For running unit-tests live while editing code, it's plausible because we can compile everything, run all tests and collect coverage, and then know which tests cover which code (though even that would be relying on the tests being deterministic). But I don't think you can reliably do the analysis which tests depend on which code statically, so this wouldn't help you reduce CI times</p>



<a name="217886407"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217886407" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217886407">(Nov 25 2020 at 13:51)</a>:</h4>
<p>right, this is dynamic - you need state between CI runs</p>



<a name="217886487"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217886487" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217886487">(Nov 25 2020 at 13:52)</a>:</h4>
<p>to know which tests depend on which code, and also which files have been modified</p>



<a name="217886702"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217886702" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217886702">(Nov 25 2020 at 13:54)</a>:</h4>
<p>hmm, I didn't think about non-determinism</p>



<a name="217886732"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217886732" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Diebold <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217886732">(Nov 25 2020 at 13:54)</a>:</h4>
<p>even then -- if your IDE live-testing doesn't run some test it should, it's a slight annoyance. If the CI doesn't run some test it should, that's a huge problem</p>



<a name="217886784"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217886784" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217886784">(Nov 25 2020 at 13:55)</a>:</h4>
<p>With lazy compilation instrumenting would be trivial. It wouldn't even be necessary to add instrumentation calls in the jitted code. Just record the function call when you lazily compile a function.</p>



<a name="217886863"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217886863" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Diebold <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217886863">(Nov 25 2020 at 13:55)</a>:</h4>
<p><span aria-label="thinking" class="emoji emoji-1f914" role="img" title="thinking">:thinking:</span> how would that work if you're running multiple tests -- you wouldn't know that the second test also called the same function, would you?</p>



<a name="217887025"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217887025" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217887025">(Nov 25 2020 at 13:57)</a>:</h4>
<p>You could reset the GOT used for swapping all calls from the lazy compilation stub to the jitted code back to the lazy compilation stubs after every test and then as optimization keep a pointer to the jitted code in a side-table.</p>



<a name="217888331"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217888331" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217888331">(Nov 25 2020 at 14:06)</a>:</h4>
<p>Opened <a href="https://github.com/bjorn3/rustc_codegen_cranelift/issues/1113">bjorn3/rustc_codegen_cranelift#1113</a> for the instrumentation.</p>



<a name="217893129"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217893129" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eggyal <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217893129">(Nov 25 2020 at 14:45)</a>:</h4>
<p><span class="user-mention" data-user-id="133247">@bjorn3</span> I guess you can do it at the function level, but instrumenting code paths within each function provides much richer feedback: eg test coverage and failure paths</p>



<a name="217896807"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217896807" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217896807">(Nov 25 2020 at 15:12)</a>:</h4>
<p><span class="user-mention" data-user-id="249694">@eggyal</span> True, but it also has much higher overhead. When trying to use cg_clif as frontend for yorick a while ago, I had to instrument the start of each basic block. When the instrumentation function immediately returned without doing anything, the overhead was 15% in the instrumentation function. Which doesn't account for any pessimizations of regalloc. When instrumentation was disabled at runtime using a global flag, the overhead was 30% after writing inline asm to prevent cranelift from spilling registers clobbered in the enabled case. Maybe the overhead could be improved by being  smarter about where to add the instrumentation calls. One instrumentation call at the top of each function may be more doable.</p>



<a name="217896937"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217896937" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217896937">(Nov 25 2020 at 15:14)</a>:</h4>
<p>Or maybe only instrument user code and keep all code in dependencies uninstrumented?</p>



<a name="217904439"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217904439" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eggyal <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217904439">(Nov 25 2020 at 16:10)</a>:</h4>
<p>Thanks <span class="user-mention" data-user-id="133247">@bjorn3</span>, that's definitely significant.  However I'd observe from using Wallaby that, after the initial run of the full test suite, one rarely sees more than a few unit tests invoked by any given edit... and, being unit tests, execution typically is of the order of milliseconds—so even a 30% uplift may not be very material in practice?</p>



<a name="217910201"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/217910201" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#217910201">(Nov 25 2020 at 16:50)</a>:</h4>
<p><span class="user-mention" data-user-id="249694">@eggyal</span> The overhead may not be very important in this case. Also thinking about it some more, the overhead can be significantly reduced in this case by simply having a global for every instrumentation point and writing directly to that global without doing a call each time.</p>



<a name="218428191"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/218428191" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#218428191">(Dec 01 2020 at 14:26)</a>:</h4>
<p>I actually don't think we need any fancy techniques here</p>



<a name="218428235"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/218428235" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#218428235">(Dec 01 2020 at 14:26)</a>:</h4>
<p>We can just re-use our <code>checkOnSave</code> infra, but with using <code>cargo test</code> instead of <code>cargo check</code></p>



<a name="218428317"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/218428317" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#218428317">(Dec 01 2020 at 14:27)</a>:</h4>
<p>I think the most work here is getting the UI bits (which are not part of standard LSP, and as such would require a bunch of custom code)</p>



<a name="218428568"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/218428568" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#218428568">(Dec 01 2020 at 14:29)</a>:</h4>
<p><span class="user-mention" data-user-id="133169">@matklad</span> that would rerun <em>all</em> the tests, right? We've been discussing how to only rerun changed tests.</p>



<a name="218428624"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/218428624" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#218428624">(Dec 01 2020 at 14:29)</a>:</h4>
<p>But I agree an opt-in way to rerun everything is good in the meantime, since incremental retesting is a bit of a research project</p>



<a name="218428846"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/218428846" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#218428846">(Dec 01 2020 at 14:31)</a>:</h4>
<p>The discussion converged to that, but we don't need that to implement the original feature request</p>



<a name="223307457"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/223307457" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eggyal <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#223307457">(Jan 19 2021 at 22:44)</a>:</h4>
<p>Just revisiting this a bit... <span class="user-mention" data-user-id="133169">@matklad</span> you mentioned most the work would be on UI as reporting test results is not in standard LSP, but VS does have Live Unit Testing for other languages so I guess MS have an API for it even if it’s not public/open?  Is there any way we could find out more, rather than (as you say) reinvent the wheel (which appears to be Wallaby’s approach)?</p>



<a name="223308575"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/223308575" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#223308575">(Jan 19 2021 at 22:55)</a>:</h4>
<p>I think just thoroughly googing around and reading the sources of other extensions would do the trick</p>



<a name="223417125"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/223417125" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> David Barsky <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#223417125">(Jan 20 2021 at 18:49)</a>:</h4>
<p>(lurking). If I understand this discussion right, VSCode has an issue for standardizing the test interface: <a href="https://github.com/microsoft/vscode/issues/107467">https://github.com/microsoft/vscode/issues/107467</a></p>



<a name="223417161"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/223417161" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> David Barsky <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#223417161">(Jan 20 2021 at 18:49)</a>:</h4>
<p>(it's part of the January 2021 iteration plan: <a href="https://github.com/microsoft/vscode/issues/112419">https://github.com/microsoft/vscode/issues/112419</a>)</p>



<a name="233637982"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/233637982" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eggyal <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#233637982">(Apr 08 2021 at 11:40)</a>:</h4>
<p>My mind was just wandering over this once again, and on re-reading the thread I think it may be worth adding to what <span class="user-mention silent" data-user-id="133247">bjorn3</span> <a href="#narrow/stream/185405-t-compiler.2Frust-analyzer/topic/Live.20unit.20testing/near/217910201">said</a>:</p>
<blockquote>
<p>simply having a global for every instrumentation point and writing directly to that global without doing a call each time.</p>
</blockquote>
<p>I believe this is exactly how <code>-Zinstrument-coverage</code> works, albeit that the counters are incremented via LLVM intrinsics that obviously aren't available in cg-clif; furthermore that approach does not instrument every block as the counts for many can be calculated from those of others (eg in <code>if ... else</code> only one branch need be counted as the other is simply the difference between the parent and that counted one).  Might it be worth adding similar instrumentation intrinsics to cg-clif?  That feels like something I could take a crack at.</p>



<a name="233642818"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/233642818" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#233642818">(Apr 08 2021 at 12:22)</a>:</h4>
<p><span class="user-mention" data-user-id="249694">@eggyal</span> For determining which functions are used, only per-function instrumentation is necessary, not per-block instrumentation. I do see value in full <code>-Zinstrument-coverage</code> support (maybe even compatible with LLVM), but for now I think per-function coverage should be easier. I am happy with a PR for either option.</p>



<a name="233646643"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Live%20unit%20testing/near/233646643" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eggyal <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Live.20unit.20testing.html#233646643">(Apr 08 2021 at 12:52)</a>:</h4>
<p>Created <a href="https://github.com/rust-analyzer/rust-analyzer/issues/8420">https://github.com/rust-analyzer/rust-analyzer/issues/8420</a> to track this</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>